Metamorphic testing (MT) is a property-based

software testing Software testing is the act of examining the artifacts and the behavior of the software under test by validation and verification. Software testing can also provide an objective, independent view of the software to allow the business to apprecia ...

technique, which can be an effective approach for addressing the

test oracle In computing, software engineering, and software testing, a test oracle (or just oracle) is a mechanism for determining whether a test has passed or failed. The use of oracles involves comparing the output(s) of the system under test, for a given ...

problem and test case generation problem. The test oracle problem is the difficulty of determining the expected outcomes of selected

test case In software engineering, a test case is a specification of the inputs, execution conditions, testing procedure, and expected results that define a single test to be executed to achieve a particular software testing objective, such as to exercise ...

s or to determine whether the actual outputs agree with the expected outcomes. ''Metamorphic relations'' (MRs) are necessary properties of the intended functionality of the software, and must involve ''multiple'' executions of the software. Consider, for example, a program that implements sin ''x'' correct to 100

significant figures Significant figures (also known as the significant digits, ''precision'' or ''resolution'') of a number in positional notation are digits in the number that are reliable and necessary to indicate the quantity of something. If a number expre ...

; a metamorphic relation for sine functions is " sin (''π − x'') = sin ''x''". Thus, even though the expected value of sin ''x''₁ for the source test case ''x''₁ = 1.234 correct to the required accuracy is not known, a follow-up test case ''x''₂ = ''π'' − 1.234 can be constructed. We can verify whether the actual outputs produced by the program under test from the source test case and the follow-up test case are consistent with the MR in question. Any inconsistency (after taking rounding errors into consideration) indicates a

failure Failure is the state or condition of not meeting a desirable or intended objective (goal), objective, and may be viewed as the opposite of Success (concept), success. The criteria for failure depends on context, and may be relative to a parti ...

of the program, caused by a fault in the implementation. MRs are not limited to programs with numerical inputs or

equality Equality may refer to: Society * Political equality, in which all members of a society are of equal standing ** Consociationalism, in which an ethnically, religiously, or linguistically divided state functions by cooperation of each group's elite ...

relations. As an example, when testing a booking website, a

web search Web most often refers to: * Spider web, a silken structure created by the animal * World Wide Web or the Web, an Internet-based hypertext system Web, WEB, or the Web may also refer to: Computing * WEB, a literate programming system created by ...

for accommodation in Sydney, Australia, returns 1,671 results; are the results of this search correct and complete? This is a test oracle problem. Based on a metamorphic relation, we may filter the price range or star rating and apply the search again; it should return a subset of the previous results. A violation of this expectation would similarly reveal a failure of the system. Metamorphic testing was invented by T.Y. Chen in the technical report in 1998. Since then, more than 150 international researchers and practitioners have applied the technique to real-life applications. Some examples include web services,

computer graphics Computer graphics deals with generating images with the aid of computers. Today, computer graphics is a core technology in digital photography, film, video games, cell phone and computer displays, and many specialized applications. A great de ...

embedded systems An embedded system is a computer system—a combination of a computer processor, computer memory, and input/output peripheral devices—that has a dedicated function within a larger mechanical or electronic system. It is ''embedded'' as ...

simulation A simulation is the imitation of the operation of a real-world process or system over time. Simulations require the use of Conceptual model, models; the model represents the key characteristics or behaviors of the selected system or proc ...

and

modeling A model is an informative representation of an object, person or system. The term originally denoted the plans of a building in late 16th-century English, and derived via French and Italian ultimately from Latin ''modulus'', a measure. Models c ...

machine learning Machine learning (ML) is a field of inquiry devoted to understanding and building methods that 'learn', that is, methods that leverage data to improve performance on some set of tasks. It is seen as a part of artificial intelligence. Machine ...

decision support A decision support system (DSS) is an information system that supports business or organizational decision-making activities. DSSs serve the management, operations and planning levels of an organization (usually mid and higher management) and ...

bioinformatics Bioinformatics () is an interdisciplinary field that develops methods and software tools for understanding biological data, in particular when the data sets are large and complex. As an interdisciplinary field of science, bioinformatics combi ...

components Circuit Component may refer to: •Are devices that perform functions when they are connected in a circuit. In engineering, science, and technology Generic systems * System components, an entity with discrete structure, such as an assem ...

numerical analysis Numerical analysis is the study of algorithms that use numerical approximation (as opposed to symbolic computation, symbolic manipulations) for the problems of mathematical analysis (as distinguished from discrete mathematics). It is the study of ...

, and

compilers In computing, a compiler is a computer program that translates computer code written in one programming language (the ''source'' language) into another language (the ''target'' language). The name "compiler" is primarily used for programs that ...

. The first major survey of the field of MT was conducted in 2016. It was followed by another major survey in 2018, which highlights the challenges and opportunities and clarifies common misunderstandings. Although MT was initially proposed as a

software verification Software verification is a discipline of software engineering whose goal is to assure that software fully satisfies all the expected requirements. Broad scope and classification A broad definition of verification makes it equivalent to software t ...

technique, it was later developed into a

paradigm In science and philosophy, a paradigm () is a distinct set of concepts or thought patterns, including theories, research methods, postulates, and standards for what constitute legitimate contributions to a field. Etymology ''Paradigm'' comes f ...

that covers verification,

validation Validation may refer to: * Data validation, in computer science, ensuring that data inserted into an application satisfies defined formats and other input criteria * Forecast verification, validating and verifying prognostic output from a numerica ...

, and other types of

software quality In the context of software engineering, software quality refers to two related but distinct notions: * Software functional quality reflects how well it complies with or conforms to a given design, based on functional requirements or specification ...

assessment. MT can be applied independently, and can also be combined with other

static Static may refer to: Places *Static Nunatak, a nunatak in Antarctica United States * Static, Kentucky and Tennessee *Static Peak, a mountain in Wyoming **Static Peak Divide, a mountain pass near the peak Science and technology Physics *Static el ...

and dynamic software analysis techniques such as proving and

debugging In computer programming and software development, debugging is the process of finding and resolving '' bugs'' (defects or problems that prevent correct operation) within computer programs, software, or systems. Debugging tactics can involve in ...

. In August 2018,

Google Google LLC () is an American multinational technology company focusing on search engine technology, online advertising, cloud computing, computer software, quantum computing, e-commerce, artificial intelligence, and consumer electronics. ...

acquired GraphicsFuzz, a startup from

Imperial College London Imperial College London (legally Imperial College of Science, Technology and Medicine) is a public research university in London, United Kingdom. Its history began with Prince Albert, consort of Queen Victoria, who developed his vision for a cu ...

, to apply metamorphic testing to graphics

device drivers In computing, a device driver is a computer program that operates or controls a particular type of device that is attached to a computer or automaton. A driver provides a software interface to hardware devices, enabling operating systems and ot ...

for Android

smartphones A smartphone is a portable computer device that combines mobile telephone and computing functions into one unit. They are distinguished from feature phones by their stronger hardware capabilities and extensive mobile operating systems, which ...

References

External links

* * * {{Cite journal, title=Automated testing of real-life self-driving systems and beyond: Keynote speech by Z.Q. (George) Zhou at IEEE AITest, San Francisco East Bay, 2019, doi = 10.5281/zenodo.2636979, year = 2019, last1 = Zhou, first1 = Zhi Quan Software testing